Large-scale industrial recommender systems are usually confronted with computational problems due to the enormous corpus size. To retrieve and recommend the most relevant items to users under response time limits, resorting to an efficient index structure is an effective and practical solution. The previous work Tree-based Deep Model (TDM) \cite{zhu2018learning} greatly improves recommendation accuracy using tree index. By indexing items in a tree hierarchy and training a user-node preference prediction model satisfying a max-heap like property in the tree, TDM provides logarithmic computational complexity w.r.t. the corpus size, enabling the use of arbitrary advanced models in candidate retrieval and recommendation.

The results presented for this work beats the benchmarks by a good deal, and in particular, the online test results are very good. It is an incremental improvement to an existing model (TDM) by doing an additional optimization step. The resulting improvement is impressive, though, and it feels like this would be more applicable to an applied data science conference such as KDD or WWW. The explanation of TDM in Section 2.1 is helpful, but it would be even more helpful to have a direct comparison between the tree building steps between TDM and the new proposed method. For example, having a side-by-side comparison of Algorithms1 & 2 with its TDM predecessor would go a long way in understanding detailed differences.

The review scores were somewhat borderline, but overall slightly above the acceptance threshold. There was some disagreement among the reviewers, following which a discussion was initiated. The rebuttal largely addresses the concerns of R1 (the most negative review), and in the metareviewer's opinion does a reasonable job of addressing these concerns, which are mostly clarifications regarding the performance of the algorithm. Positively, the reviewers mostly concur that the method, while fairly straightforward, offers significant improvements over existing techniques. After discussion there was some positive movement in review scores resulting in a positive consensus among reviewers.

Large-scale industrial recommender systems are usually confronted with computational problems due to the enormous corpus size. To retrieve and recommend the most relevant items to users under response time limits, resorting to an efficient index structure is an effective and practical solution. The previous work Tree-based Deep Model (TDM) \cite{zhu2018learning} greatly improves recommendation accuracy using tree index. By indexing items in a tree hierarchy and training a user-node preference prediction model satisfying a max-heap like property in the tree, TDM provides logarithmic computational complexity w.r.t. the corpus size, enabling the use of arbitrary advanced models in candidate retrieval and recommendation.